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NETWORK SURVEILLANCE 

REFERENCE TO GOVERNMENT FUNDING 

This invention was made with Government support under 
Contract Number F30602-96-C-0294 awarded by DARPA. 5 
The Government has certain rights in this invention. 

REFERENCE TO APPENDIX 

A microfiche appendix is included as part of the specifi- 10 
cation. The microfiche appendix includes material subject to 
copyright protection. The copyright owner does not object to 
the reproduction of the microfiche appendix, as it appears in 
the Patent and Trademark Office patent file or records, but 
otherwise reserves all copyright rights. This application 15 
contains Microfiche Appendix containing ten (10) slides and 
956 frames. 

BACKGROUND 

The invention relates to computer networks. 20 
Computer networks offer users ease and efficiency in 
exchanging information. Networks tend to include conglom- 
erates of integrated commercial and custom-made 
components, in terope rating and sharing information at 
increasing levels of demand and capacity. Such varying 25 
networks manage a growing list of needs including 
transportation, commerce, energy management, 
communications, and defense. 

Unfortunately, the very interoperability and sophisticated 3fl 
integration of technology that make networks such valuable 
assets also make them vulnerable to attack, and make 
dependence on networks a potential liability. Numerous 
examples of planned network attacks, such as the Internet 
worm, have shown how interconnectivity can be used to ^ 
spread harmful program code. Accidental outages such as 
the 1980 ARPAnet collapse and the 1990 AT&T collapse 
illustrate how seemingly localized triggering events can 
have globally disastrous effects on widely distributed sys- 
tems. In addition, organized groups have performed mali- ^ 
cious and coordinated attacks against various online targets. 

SUMMARY 

In general, in one aspect, a method of network surveil- 
lance includes receiving network packets (e.g., TCP/IP 4S 
packets) handled by a network entity and building at least 
one long-term and at least one short-term statistical profile 
from at least one measure of the network packets that 
monitors data transfers, errors, or network connections. A 
comparison of at least one long-term and at least one 50 
short-term statistical profile is used to determine whether the 
difference between the short-term statistical profile and the 
long-term statistical profile indicates suspicious network 
activity. 

Embodiments may include one or more of the following 55 
features. The measure may monitor data transfers by moni- 
toring network packet data transfer commands, data transfer 
errors, and/or monitoring network packet data transfer vol- 
ume. The measure may monitor network connections by 
monitoring network connection requests, network connec- go 
tion denials, and/or a correlation of network connections 
requests and network connection denials. The measure may 
monitor errors by monitoring error codes included in a 
network packet such as privilege error codes and/or error 
codes indicating a reason a packet was rejected. 55 

The method may also include responding based on the 
determining whether the difference between a short-term 
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statistical profile and a long-term statistical profile indicates 
suspicious network activity. A response may include altering 
analysis of network packets and/or severing a communica- 
tion channel. A response may include transmitting an event 
record to a network monitor, such as hierarchically higher 
network monitor and/or a network monitor that receives 
event records from multiple network monitors. 

The network entity may be a gateway, a router, or a proxy 
server. The network entity may instead be a virtual private 
network entity (e.g., node). 

In general, in another aspect, a method of network sur- 
veillance includes monitoring network packets handled by a 
network entity and building a long-term and multiple short- 
term statistical profiles of the network packets. A compari- 
son of one of the multiple short-term statistical profiles with 
the long-term statistical profile is used to determine whether 
the difference between the short-term statistical profiles and 
the long-term statistical profile indicates suspicious network 
activity. 

Embodiments may include one or more of the following. 
The multiple short-term statistical profiles may monitor 
different anonymous FTP sessions. Building multiple short- 
term statistical profiles may include deinterleaving packets 
to identify a short-term statistical profile. 

In general, in another aspect, a computer program 
product, disposed on a computer readable medium, includes 
instructions for causing a processor to receive network 
packets handled by a network entity and to build at least one 
long-term and at least one short-term statistical profile from 
at least one measure of the network packets that monitors 
data transfers, errors, or network connections. The instruc- 
tions compare a short-term and a long-term statistical profile 
to determine whether the difference between the short-term 
statistical profile and the long-term statistical profile indi- 
cates suspicious network activity. 

In general, in another aspect, a method of network sur- 
veillance includes receiving packets at a virtual private 
network entity and statistically analyzing the received pack- 
ets to determine whether the packets indicate suspicious 
network activity. The packets may or may not be decrypted 
before statistical analysis. 

Advantages may include one or more of the following. 
Using long-term and a short-term statistical profiles from 
measures that monitor data transfers, errors, or network 
connections protects network components from intrusion. 
As long-term profiles represent "normal" activity, abnormal 
activity may be detected without requiring an administrator 
to catalog each possible attack upon a network. Additionally, 
the ability to deinterleave packets to create multiple short- 
term profiles for comparison against a long-term profile 
enables the system to detect abnormal behavior that may be 
statistically ameliorated if only a single short-term profile 
was created. 

The scheme of communication network monitors also 
protects networks from more global attacks. For example, an 
attack made upon one network entity may cause other 
entities to be alerted. Further, a monitor that collects event 
reports from different monitors may correlate activity to 
identify attacks causing disturbances in more than one 
network entity. 

Additionally, statistical analysis of packets handled by a 
virtual private network enable detection of suspicious net- 
work activity despite virtual private network security tech- 
niques such as encryption of the network packets. 

Other features and advantages will become apparent from 
the following description, including the drawings, and from 
the claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a diagram of network monitors deployed in an 
enterprise. 

FIG. 2 is a diagram of a network monitor that monitors an 
event stream. 

FIG. 3 is a diagram of a resource object that configures the 
network monitor of FIG. 2. 

FIG. 4 is a flowchart illustrating network surveillance. 

FIG. 5 is a flowchart illustrating multiple short-term 
statistical profiles for comparison against a single long-term 
statistical profile. 

FIG. 6 is a diagram of a computer platform suitable for 
deployment of a network monitor. 

DETAILED DESCRIPTION 

Referring to FIG. 1, an enterprise 10 includes different 
domains 12o-12c. Each domain 12«-12c includes one or 
more computers offering local and network services that 
provide an interface for requests internal and external to the 
domain Xla-Xlc. Network services include features com- 
mon to many network operating systems such as mail, 
HTTP, FTP, remote login, network file systems, finger, 
Kerberos, and SNMP. Some domains Xla-Xlc may share 
trust relationships with other domains (either peer-to-peer or 
hierarchical). Alternatively, domains I2a-12e may operate 
in complete mistrust of all others, providing outgoing con- 
nections only or severely restricting incoming connections. 
Users may be local to a single domain or may possess 
accounts on multiple domains that allow them to freely 
establish connections throughout the enterprise 10. 

As shown, the enterprise 10 includes dynamically 
deployed network monitors 16^-16/ that analyze and 
respond to network activity and can interoperate to form an 
analysis hierarchy. The analysis hierarchy provides a frame- 
work for the recognition of more global threats to interdo- 
main connectivity, including coordinated attempts to infil- 
trate or destroy connectivity across an entire network 
enterprise 10. The hierarchy includes service monitors 
16ff-16c, domain monitors 16d-l Se, and enterprise moni- 
tors 16/ 

Service monitors I6a-16c provide local real-time analy- 
sis of network packets (e.g., TCP/IP packets) handled by a 
network entity 14a-14c. Network entities include gateways, 
routers, firewalls, or proxy servers. A network entity may 
also be part of a virtual private network. A virtual private 
network (VPN) is constructed by using public wires to 
connect nodes. For example, a network could use the 
Internet as the medium for transporting data and use encryp- 
tion and other security mechanisms to ensure that only 
authorized users access the network and that the data cannot 
be intercepted. A monitor 16a-16/can analyze packets both 
before and after decryption by a node of the virtual private 
network. 

Information gathered by a service monitor 16o-16c can 
be disseminated to other monitors 16a-16/, for example, via 
a subscription-based communication scheme. In a 
subscription-based scheme client monitors subscribe to 
receive analysis reports produced by server monitors. As a 
monitor X6a-~X6f produces analysis reports, the monitor 
16a-16/ disseminates these reports asynchronously to sub- 
scribers. Through subscription, monitors 16a-16/ distrib- 
uted throughout a large network are able to efficiently 
disseminate reports of malicious activity without requiring 
the overhead of synchronous polling. 

Domain monitors 16d-X6e perform surveillance over all 
or part of a domain Xla-Ylc. Domain monitors 16d-16e 
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correlate intrusion reports disseminated by individual ser- 
vice monitors 16a-16c, providing a domain- wide perspec- 
tive of activity (or patterns of activity). In addition to domain 
surveillance, domain monitors 16a-16c can reconfigure 

5 system parameters, interface with other monitors beyond a 
domain, and report threats against a domain Xla-Xlc to 
administrators. Domain monitors 16d-16e can subscribe to 
service monitors 16<z-16c. Where mutual trust among 
domains Xla-Xlc exists, domain monitors I6d-16e may 

10 establish peer relationships with one another. Peer-to-peer 
subscription allows domain monitors X6d-16e to share 
analysis reports produced in other domains Xla-Xlc. 
Domain monitors 16d-16e may use such reports to dynami- 
cally sensitize their local service monitors 16a-16c to mali- 

15 cious activity found to be occurring outside a domain 
12a-12c. Domain monitors 16d—X6e may also operate 
within an enterprise hierarchy where they disseminate analy- 
sis reports to enterprise monitors 16/ for global correlation. 
Enterprise monitors 16/ correlate activity reports pro- 

20 duced across the set of monitored domains 12a— 12c. Enter- 
prise 10 surveillance may be used where domains \la-X2c 
are interconnected under the control of a single organization, 
such as a large privately owned WAN (Wide Area Network). 
The enterprise 10, however, need not be stable in its con- 

25 figuration or centrally administered. For example, the enter- 
prise 10 may exist as an emergent entity through new 
interconnections of domains 12a— Vic. Enterprise 10 surveil- 
lance is very similar to domain X2a-\2c surveillance: an 
enterprise monitor 16/ subscribes to various domain moni- 

30 tors 16d-16e, just as the domain monitors 16d-16e sub- 
scribed to various service monitors 16a-16c. The enterprise 
monitor 16/ (or monitors, as it would be important to avoid 
centralizing any analysis) focuses on network-wide threats 
such as Internet worm-like attacks, attacks repeated against 

35 common network services across domains, or coordinated 
attacks from multiple domains against a single domain. As 
an enterprise monitor 16/ recognizes commonalities in intru- 
sion reports across domains (e.g., the spreading of a worm 
or a mail system attack repeated throughout the enterprise 

40 10), the monitor 16/can help domains lla-Xlc counter the 
attack and can sensitize other domains 12a-12c to such 
attacks before they are affected. Through correlation and 
sharing of analysis reports, reports of problems found by one 
monitor 16^-16/ may propagate to other monitors 16^-16/ 

45 throughout the network. Interdomain event analysis is vital 
to addressing more global, information attacks against the 
entire enterprise 10. 

Referring to FIG. 2, each monitor 16 includes one or more 
analysis engines 22, 24. These engines 22, 24 can be 

50 dynamically added, deleted, and modified as necessary. In 
the dual-analysis configuration shown, a monitor 16 instan- 
tiation includes a signature analysis engine 22 and a statis- 
tical profiling engine 24. In general, a monitor 16 may 
include additional analysis engines that may implement 

55 other forms of analysis. A monitor 16 also includes a 
resolver 20 that implements a response policy and a resource 
object 32 that configures the monitor 16. The monitors 16 
incorporate an application programmers* interface (API) 
that enhances encapsulation of monitor functions and eases 

60 integration of third-party intrusion-detection tools 28, 30. 
Each monitor 16 can analyze event records that form an 
event stream. The event stream may be derived from a 
variety of sources such as TCP/IP network packet contents 
or event records containing analysis reports disseminated by 

65 other monitors. For example, an event record can be formed 
from data included in the header and data segment of a 
network packet. The volume of packets transmitted and 
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received, however, dictates careful assessment of ways to 
select and organize network packet information into event 
record streams. 

Selection of packets can be based on different criteria. 
Streams of event records can be derived from discarded 
traffic (i.e., packets not allowed through the gateway because 
they violate filtering rules), pass-through traffic (i.e., packets 
allowed into the internal network from external sources), 
packets having a common protocol (e.g., all ICMP (Internet 
Control Message Protocol) packets that reach the gateway), 
packets involving network connection management (e.g., 
SYN, RESET, ACK, [window resize]), and packets targeting 
ports to which an administrator has not assigned any net- 
work service and that also remain unblocked by the firewall. 
Event streams may also be based on packet source addresses 
(e.g., packets whose source addresses match well-known 
external sites such as satellite offices or have raised suspi- 
cion from other monitoring efforts) or destination addresses 
(e.g., packets whose destination addresses match a given 
internal host or workstation). Selection can also implement 
application- layer monitoring (e.g., packets targeting a par- 
ticular network service or application). Event records can 
also be produced from other sources of network packet 
information such as report logs produced by network enti- 
ties. Event streams can be of very fine granularity. For 
example, a different stream might be derived for commands 
received from different commercial web-browsers since 
each web-browser produces different characteristic network 
activity. 

A monitor 16 can also construct interval summary event 
records, which contain accumulated network traffic statistics 
(e.g., number of packets and number of kilobytes 
transferred). These event records are constructed at the end 
of each interval (e.g., once per N seconds). Event records are 
forwarded to the analysis engines 22, 24 for analysis. 

The profile engine 22 can use a wide range of multivariate 
statistical measures to profile network activity indicated by 
an event stream. A statistical score represents how closely 
currently observed usage corresponds to the established 
patterns of usage. The profiler engine 22 separates profile 
management and the mathematical algorithms used to assess 
the anomaly of events. The profile engine 22 may use a 
statistical analysis technique described in A. Valdes and D. 
Anderson, "Statistical Methods for Computer Usage 
Anomaly Detection Using NIDES", Proceedings of the 
Third International Workshop on Rough Sets and Soft 
Computing, January 1995, which is incorporated by refer- 
ence in its entirety. Such ao engine 22 can profile network 
activity via one or more variables called measures. Measures 
can be categorized into four classes: categorical, continuous, 
intensity, and event distribution measures. 

Categorical measures assume values from a discrete, 
nonordered set of possibilities. Examples of categorical 
measures include network source and destination addresses, 
commands (e.g., commands that control data transfer and 
manage network connections), protocols, error codes (e.g., 
privilege violations, malformed service requests, and mal- 
formed packet codes), and port identifiers. The profiler 
engine 22 can build empirical distributions of the category 
values encountered, even if the list of possible values is 
open-ended. The engine 22 can have mechanisms for "aging 
out" categories whose long-term probabilities drop below a 
threshold. 

Continuous measures assume values from a continuous or 
ordinal set. Examples include inter-event time (e.g., differ- 
ence in time stamps between consecutive events from the 
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same stream), counting measures such as the number of 
errors of a particular type observed in the recent past, the 
volume of data transfers over a period of time, and network 
traffic measures (number of packets and number of 

5 kilobytes). The profiler engine 22 treats continuous mea- 
sures by first allocating bins appropriate to the range of 
values of the underlying measure, and then tracking the 
frequency of observation of each value range. In this way, 
multi-modal distributions are accommodated and much of 

1Q the computational machinery used for categorical measures 
is shared. Continuous measures are useful not only for 
intrusion detection, but also to support the monitoring of the 
health and status of the network from the perspective of 
connectivity and throughput. For example, a measure of 

15 traffic volume maintained can detect an abnormal loss in the 
data rate of received packets when this volume falls outside 
historical norms. This sudden drop can be specific both to 
the network entity being monitored and to the time of day 
(e.g., the average sustained traffic rate for a major network 

2Q artery is much different at 11:00 a.m. than at midnight). 
Intensity measures reflect the intensity of the event stream 
(e.g., number of ICMP packets) over specified time intervals 
(e.g., 1 minute, 10 minutes, and 1 hour). Intensity measures 
are particularly suited for detecting flooding attacks, while 

25 also providing insight into other anomalies. 

Event distribution measures are meta-measures that 
describes how other measures in the profile are affected by 
each event. For example, an "Is" command in au FTP 
session affects the directory measure, but does not affect 

30 measures related to file transfer. This measure is not inter- 
esting for all event streams. For example, all network-traffic 
event records affect the same measures (number of packets 
and kilobytes) defined for that event stream, so the event 
distribution does not change. On the other hand, event 

35 distribution measures are useful in correlative analysis per- 
formed by a monitor 16a-16/ that receives reports from 
other monitors 16a-16/ 

The system maintains and updates a description of behav- 
ior with respect to these measure types in an updated profile. 

40 The profile is subdivided into short-term and long-term 
profiles. The short-term profile accumulates values between 
updates, and exponentially ages (e.g., weighs data based on 
how long ago the data was collected) values for comparison 
to the long-term profile. As a consequence of the aging 

45 mechanism, the short-term profile characterizes recent 
activity, where "recent" is determined by a dynamically 
configurable aging parameters. At update time (typically, a 
time of low system activity), the update function folds the 
short-term values observed since the last update into the 

50 long-term profile, and the short-term profile is cleared. The 
long-term profile is itself slowly aged to adapt to changes in 
subject activity. Anomaly scoring compares related 
attributes in the short-term profile against the long-term 
profile. As all evaluations are done against empirical 

ss distributions, no assumptions of parametric distributions are 
made, and multi-modal and categorical distributions are 
accommodated. Furthermore, the algorithms require no a 
priori knowledge of intrusive or exceptional activity. 
The statistical algorithm adjusts a short-term profile for 

60 the measure values observed in the event record. The 
distribution of recently observed values is compared against 
the long-term profile, and a distance between the two is 
obtained. The difference is compared to a historically adap- 
tive deviation. The empirical distribution of this deviation is 

65 transformed to obtain a score for the event. Anomalous 
events are those whose scores exceed a historically adaptive 
score threshold based on the empirical score distribution. 
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This nonparametric approach handles all measure types and 
makes no assumptions on the modality of the distribution for 
continuous measures. 

Profiles are provided to the computational engine as 
classes defined in the resource object 32. The mathematical 
functions for anomaly scoring, profile maintenance, and 
updating do not require knowledge of the data being ana- 
lyzed beyond what is encoded in the profile class. Event 
collection interoperability supports translation of the event 
stream to the profile and measure classes. At that point, 
analysis for different types of monitored entities is math- 
ematically similar. This approach imparts great flexibility to 
the analysis in that fading memory constants, update 
frequency, measure type, and so on are tailored to the 
network entity being monitored. 

The measure types described above can be used individu- 
ally or in combination to detect network packet attributes 
characteristic of intrusion. Such characteristics include large 
data transfers (e.g., moving or downloading files), an 
increase in errors (e.g., an increase in privilege violations or 
network packet rejections), network connection activity, and 
abnormal changes in network volume. 

As shown, the monitor 16 also includes a signature engine 
24. The signature engine 24 maps an event stream against 
abstract representations of event sequences that are known 
to indicate undesirable activity. Signature-analysis objec- 
tives depend on which layer in the hierarchical analysis 
scheme the signature engine operates. Service monitor 
16a-16c signature engines 24 attempt to monitor for 
attempts to penetrate or interfere with the domain's opera- 
tion. The signature engine scans the event stream for events 
that represent attempted exploitations of known attacks 
against the service, or other activity that stands alone as 
warranting a response from the monitor. Above the service 
layer, signature engines 24 scan the aggregate of intrusion 
reports from service monitors in an attempt to detect more 
global coordinated attack scenarios or scenarios that exploit 
interdependencies among network services. Layering signa- 
ture engine analysis enables the engines 24 to avoid mis- 
guided searches along incorrect signature paths in addition 
to distributing the signature analysis. 

A signature engines 24 can detect, for example, address 
spoofing, tunneling, source routing, SATAN attacks, and 
abuse of ICMP messages ("Redirect" and "Destination 
Unreachable" messages in particular). Threshold analysis is 
a rudimentary, inexpensive signature analysis technique that 
records the occurrence of specific events and, as the name 
implies, detects when the number of occurrences of that 
event surpasses a reasonable count. For example, monitors 
can encode thresholds to monitor activity such as the num- 
ber of fingers, pings, or failed login requests to accounts 
such as guest, demo, visitor, anonymous FTP, or employees 
who have departed the company. 

Signature engine 24 can also examine the data portion of 
packets in search of a variety of transactions that indicate 
suspicious, if not malicious, intentions by an external client. 
The signature engine 24, for example, can parse FTP traffic 
traveling through the firewall or router for unwanted trans- 
fers of configuration or specific system data, or anonymous 
requests to access non-public portions of the directory 
structure. Similarly, a monitor can analyze anonymous FTP 
sessions to ensure that the file retrievals and uploads/ 
modifications are limited to specific directories. 
Additionally, signature analysis capability can extend to 
session analyses of complex and dangerous, but highly 
useful, services like HTTP or Gopher. 
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Signature analysis can also scan traffic directed at unused 
ports (i.e., ports to which the administrator has not assigned 
a network service). Here, packet parsing can be used to study 
network traffic after some threshold volume of traffic, 
5 directed at an unused port, has been exceeded. A signature 
engine 24 can also employ a knowledge base of known 
telltale packets that are indicative of well-known network- 
service protocol traffic (e.g., FTP, Telnet, SMTP, HTTP). 
The signature engine 24 then determines whether the 
unknown port traffic matches any known packet sets. Such 
comparisons could lead to the discovery of network services 
that have been installed without an administrator's knowl- 
edge. 

The analysis engines 22, 24 receive large volumes of 

15 events and produce smaller volumes of intrusion or suspi- 
cion reports that are then fed to the resolver 20. The resolver 
20 is an expert system that receives the intrusion and 
suspicion reports produced by the analysis engines 22, 24 
and reports produced externally by other analysis engines to 

20 which it subscribes. Based on these reports, the resolver 20 
invokes responses. Because the volume of intrusion and 
suspicion reports is lower than the volume of events 
received by the analysis engines 22, 24, the resolver 20 can 
afford the more sophisticated demands of configuration 

25 maintenance and managing the response handling and exter- 
nal interfaces necessary for monitor operation. Furthermore, 
the resolver 20 adds to extensibility by providing the sub- 
scription interface through which third-party analysis tools 
28, 30 can interact and participate in the hierarchical analy- 

30 sis scheme. 

Upon its initialization, the resolver 20 initiates authenti- 
cation and subscription sessions with those monitors 
16^-16/ whose identities appear in the monitor's 16 
subscription-list (46 FIG. 3). The resolver 20 also handles all 

3S incoming requests by subscribers, which must authenticate 
themselves to the resolver 20. Once a subscription session is 
established with a subscriber monitor, the resolver 20 acts as 
the primary interface through which configuration requests 
are received and intrusion reports are disseminated. 

40 Thus, resolve rs 20 can request and receive reports from 
other resolvers at lower layers in the analysis hierarchy. The 
resolver 20 forwards analysis reports received from sub- 
scribees to the analysis engines 22, 24. This tiered collection 
and correlation of analysis results allows monitors X6a-I6f 

45 to represent and profile global malicious or anomalous 
activity that is not visible locally. 

In addition to external- interface responsibilities, the 
resolver 20 operates as a fully functional decision engine, 
capable of invoking real-time response measures in response 

so to malicious or anomalous activity reports produced by the 
analysis engines. The resolver 20 also operates as the center 
of intramonitor communication. As the analysis engines 22, 
24 build intrusion and suspicion reports, they propagate 
these reports to the resolver 20 for further correlation, 

55 response, and dissemination to other monitors 16a-16/. The 
resolver 20 can also submit runtime configuration requests 
to the analysis engines 22, 24, for example, to increase or 
decrease the scope of analyses (e.g., enable or disable 
additional signature rules) based on various operating met- 

60 rics. These configuration requests could be made as a result 
of encountering other intrusion reports from other subscrib- 
ers. For example, a report produced by a service monitor 
16a-16c in one domain could be propagated to an enterprise 
monitor 16/, which in turn sensitizes service monitors in 

65 other domains to the same activity. 

The resolver 20 also operates as the interface mechanism 
between administrators and the monitor 16. From the per- 
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spective of a resolver 20, the administrator interface is to its parent enterprise monitor 16/. The enterprise monitor 
simply a subscribing service to which the resolver 20 may 16/ operates as a client to one or more domain monitors 
submit reports and receive configuration requests. An 16ef-16e, allowing them to correlate and model enterprise- 
administrative interface tool can dynamically subscribe and wide activity from the domain-layer results. Domain moni- 
unsubscribe to any of the deployed resolvers 20, as well as 5 tors 16d-16e operate as servers to the enterprise monitors 
submit configuration requests and asynchronous probes as 16 /> and ^ clieDls 10 ^ e se™<* monitors 16fl-16c deployed 
desired. throughout their domain 12a- 12c. This message scheme can 
^ . , ... . * operate substantially the same if correlation were to continue 
The monitors 16a-16/ incorporate a bidirectional mes- M u hef , of abstraction ^ d enterprise 10 ana i ysis . 

saeinc system that uses a standard interface specification for . ° . 4 , . t . A - 

° ° . A . , , ^ r , 4 _j i a Intramonitor and intermomtor programmmg mterfaces 

communication within and between monitor elements and 10 i_ * *• « *i_ -n. • * Zc u , 

. , . T . . . ■ « . « are substantially the same. These mterfaces can be subdi- 
external modules. Using this interface specification, third- . . . c . . c . . . i • i 
j i -»o ia vided into five categories of interoperation: channel initial- 
party modules 28, 30 can communicate with monitors. For ... . f. . i u • j 

, . . * L , , „ 0 , . A , ization and termination, channel synchronization, dynamic 

example, third-party modules 28 can submit event records to ^ . / . j* ■ 

4l _ , • • u r a j j-,- n configuration, server probmg, and report/event dissemina- 

toe analysis engmes 22, 24 for processing. Additionally, tion * aients are ini^g and terminating 

third-party modules 30 may also submit and receive analysis 15 . , ... ™. t 0 , , 

. , , ™ . , • 4 ™ . « . « channel sessions with servers. Clients are also responsible 

results via the resolver s 20 external mterfaces. Thus, third- f , , • • 4 • . .» 4 r r 

j i ->o in * *u i* £_ for managmg channel synchronization in the event of errors 

party modules 28, 30 can incorporate the results from . . 7 . , ee -i j i 

r ' ... -ii 5r . u . .i_ ■ m message sequencing or periods of failed or slow response 

monitors into other surveillance efforts or contnbute their „ T , & „ % \. x 01 . t , r , . 4 

... . i£ T . , (i.e., I m alive confirmations). Clients may also submit 

results to other monitors lfa-16/. Lasdy, the monitor s 16 ^ * . c t ; c * i 

. AriT n ... , _ ' . . J . nn dynamic configuration requests to servers. For example, an 

mternal API allows third-party analysis engmes to be linked z° ' . . • <><+ * if *• 

* . . * . . . * - t l j analysis engine 22, 24 may request an event collection 

directly into the monitor boundary. tQ * odify ^ s 7 manUcs CUents may ^ 

The message system operates under an asynchronous probe servers f or rep ort summaries or additional event 

communication model for handling results dissemination information. Lastly, servers may send clients intrusion/ 

and processing that is generically referred to as subscription- suspicion reports in response to client probes or in an 

based message passing. Component interoperation is client/ asynchronous dissemination mode. 

server-based, where a client module may subscribe to The of tQe m e m framework 

receive event data or analysis results from servers Once a specification of a transport mechanism used to 

subscription request is accepted by the server, the server establish a - communication channel betweeQ monitors 

module forwards events or analysis results to the client lfa _ 16/ of ibl betwecn a monitor 16a _ Uf ^ a 

automatically as data becomes avadable and may dynami- ^ { securit modulc ^ ^ imp i cm entation dependen- 

cally reconfigure itself as requested by the client s control dcs tfac m c s fr amework are addressed by 

requests. This asynchronous model reduces the need for pluggable tr^port mo dmes. Traiisport modules are specific 

client probes and acknowledgments. tQ the participating intrusion-detection modules, their 

The interface supports an implementation-neutral com- 35 respective hosts, and potentially to the network — should the 

munication framework that separates the programmer's modules require cross-platform interoperation. Instantiating 

interface specification and the issues of message transport. a monitor 16a-16/may involve incorporation of the neces- 

The interface specification embodies no assumptions about saiy transport module(s) (for both internal and external 

implementation languages, host platform, or a network. The communication). 

transport layer is architecturally isolated from the internals ^ ^ transport modules that handle intramonitor commu- 
of the monitors so that transport modules may be readily Nation may be different from the transport modules that 
introduced and replaced as protocols and security require- handle mt ermonitor communication. This allows the intra- 
ments are negotiated between module developers. The inter- monitor transport modules to address security and reliability 
face specification involves the definition of the messages differently than how the intermonitor transport mod- 
mat me various mtrusion-detection moohiles must convey to 4S ulcs address security and reliability. While intramonitor 
one another and how these messages should be processed. communication may more commonly involve interprocess 
The message structure and content are specified in a com- communication within a single host, intermonitor commu- 
pletely implementation-neutral context. nication will most commonly involve cross-platform net- 
Both intramonitor and intermonitor communication worked interoperation. For example, the intramonitor trans- 
employ identical subscription -based client-server models. 50 port mechanisms may employ unnamed pipes which 
With respect to intermonitor communication, the resolver 20 provides a kernel-enforced private interprocess communi- 
operates as a client to the analysis engines, and the analysis cation channel between the monitor 16 components (this 
engines 22, 24 operate as clients to the event filters. Through assumes a process hierarchy within the monitor 16 
the internal message system, the resolver 20 submits con- architecture). The monitor's 16 external transport, however, 
figuration requests to the analysis engines 22, 24, and ss will more likely export data through untrusted network 
receives from the analysis engines 22, 24 their analysis connections and thus require more extensive security man- 
results. The analysis engines 22, 24 operate as servers agement. To ensure me security and integrity of the message 
providing the resolver 20 with intrusion or suspicion reports exchange, the external transport may employ public/private 
either asynchronously or upon request. Similarly, the analy- key authentication protocols and session key exchange, 
sis engines 22, 24 are responsible for establishing and 60 Using this same interface, third -party analysis tools may 
maintaining a communication link with an event collection authenticate and exchange analysis results and configuration 
method (or event filter) and prompting the reconfiguration of information in a well-defined, secure manner, 
the collection method's filtering semantics when necessary. The pluggable transport permits flexibility in negotiating 
Intermonitor communication also operates using the security features and protocol usage with third parties, 
subscription-based hierarchy. A domain monitor 16d-16e 65 Incorporation of a commercially available network manage- 
subscribes to the analysis results produced by service moni- ment system can deliver monitoring results relating to 
tors 16a-16c, and then propagates its own analytical reports security, reliability, availability, performance, and other 
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attributes. The network management system may in turn stances under which the method should be invoked. These 

subscribe to monitor produced results in order to influence metrics include a threshold metric that corresponds to the 

network reconfiguration. measure values and scores produced by the profiler engine 

All monitors (service, domain, and enterprise) 16a-16/ 22 and severity metrics that correspond to subsets of the 

use the same monitor code-base. However, monitors may 5 associated attack sequences defined within the resource 

include different resource objects 32 having different con- object 32. 

figuration data and methods. This reusable software archi- Countermeasures range from very passive responses, such 

tecrure can reduce implementation and maintenance efforts. as report dissemination to other monitors I60-I6/ or 

Customizing and dynamically configuring a monitor 16 thus administrators, to highly aggressive actions, such as sever- 

becomes a question of building and/or modifying the 10 ing a communication channel or the reconfiguration of 

resource object 32. logging facilities within network components (e.g., routers, 

Referring to FIG. 3, the resource object 32 contains the firewalls, network services, audit daemons). An active 

operating parameters for each of the monitor's 16 compo- response may invoke handlers that validate the integrity of 

nents as well as the analysis semantics (e.g., the profiler network services or other assets to ensure that privileged 

engine's 22 measure and category definition, or the signa- 15 network services have not been subverted. Monitors 

ture engine's 24 penetration rule-base) necessary to process 16a-16/may invoke probes in an attempt to gather as much 

an event stream. After defining a resource object 32 to counterintelligence about the source of suspicious traffic by 

implement a particular set of analyses on an event stream, features such as traceroute or finger, 

the resource object 32 may be reused by other monitors 16 The resource object 32 may include a subscription list 46 

deployed to analyze equivalent event streams. For example, 20 that includes information necessary for establishing 

the resource object 32 for a domain's router may be reused subscription-based communication sessions, which may 

as other monitors 16 are deployed for other routers in a include network address information and public keys used 

domain 12a-12c. A library of resource objects 32 provides by the monitor to authenticate potential clients and servers, 

prefabricated resource objects 32 for commonly available The subscription list 46 enables transmission or reception of 

network entities. 25 messages that report malicious or anomalous activity 

The resource object 32 provides a pluggable configuration between monitors. The most obvious examples where rela- 

module for tuning the generic monitor code-base to a tionships are important involve interdependencies among 

specific event stream. The resource object 32 includes network services that make local policy decisions. For 

configurable event structures 34, analysis unit configuration example, the interdependencies between access checks per- 

38a-38/i, engine configuration 40a-40/i, resolver configu- formed during network file system mounting and the IP 

ration 42, decision unit configuration 44, subscription list mapping of the DNS service. An unexpected mount moni- 

data 46, and response methods 48. tared by the network file system service may be responded 

Configurable event structures 34 define the structure of 10 differently if the DNS monitor informs the network file 
event records and analysis result records. The monitor 35 system monitor of suspicious updates to the mount request- 
code -base maintains no internal dependence on the content or s ^NS mapping. 

or format of any given event stream or the analysis results The contents of the resource object 32 are defined and 

produced from analyzing the event stream. Rather, the utilized during monitor 16 initialization. In addition, these 

resource object 32 provides a universally applicable syntax fields may be modified by internal monitor 16 components, 

for specifying the structure of event records and analysis ^ and by authorized external clients using the monitor's 16 

results. Event records are defined based on the contents of an API. Modifying the resource object 32 permits adaptive 

event stream(s). Analysis result structures are used to pack- analysis of an event stream, however, it also introduces a 

age the findings produced by analysis engines. Event records potential stability problem if dynamic modifications are not 

and analysis results are defined similarly to allow the tightly restricted to avoid cyclic modifications. To address 

eventual hierarchical processing of analysis results as event 45 this issue, monitors 16 can be configured to accept configu- 

records by subscriber monitors. ration requests from only higher-level monitors 16. 

Event-collection methods 36 gather and parse event Referring to FIG. 4, a monitor performs network surveil- 

records for analysis engine processing. Processing by analy- lance by monitoring 66 a stream of network packets. The 

sis engines is controlled by engine configuration 40a-40/j monitor builds a statistical model of network activity from 

variables and data structures that specify the operating 50 the network packets, for example, by building 68 long-term 

configuration of a fielded monitor's analysis engine(s). The and short-term statistical profiles from measures derived 

resource object 32 maintains a separate collection of oper- from the network packets. The measures include measures 

ating parameters for each analysis engine instantiated in the that can show anomalous network activity characteristic of 

monitor 16. Analysis unit configuration 38a-38n include network intrusion such as measures that describe data 

configuration variables that define the semantics employed 55 transfers, network connections, privilege and network 

by the analysis engine to process the event stream. errors, and abnormal levels of network traffic. The monitor 

The resolver configuration 42 includes operating param- can compare 70 the long-term and short-term profiles to 

eters mat specify me coiifiguration of me ^ detect suspicious network activity. Based on this 

modules. The decision unit configuration 44 describes comparison, the monitor can respond 72 by reporting the 

semantics used by the resolver's decision unit for merging 60 activity to another monitor or by executing a countermea- 

the analysis results from the various analysis engines. The sure response. More information can be found in P. Porras 

semantics include the response criteria used to invoke coun- and A. Valdes "Live Traffic Analysis of TCP/IP Gateways", 

termeasure handlers. A resource object 32 may also include Networks and Distributed Systems Security Symposium, 

response methods 48. Response methods 48 include prepro- March 1998, which is incorporated by reference in its 

grammed countermeasure methods that the resolver may 65 entirety. 

invoke as event records are received. A response method 48 A few examples can illustrate this method of network 

includes evaluation metrics for determining the circum- surveillance. Network intrusion frequently causes large data 
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transfers, for example, when an intruder seeks to download quality or internal errors in neighboring hosts. High volumes 

sensitive files or replace system files with harmful substi- of discarded packets can also indicate more maliciously 

tutes. A statistical profile to detect anomalous data transfers intended transmissions such as scanning of UPD ports or IP 

might include a continuous measure of file transfer size, a address scanning via ICMP echoes. Excessive number of 

categorical measure of the source or destination directory of 5 ma il expansion request commands (EXPN) may indicate 

the data transfer, and an intensity measure of commands intelligence gathering, for example, by spammers. 

corresponding to data transfers (e.g., commands that down- A , . _ „. , _ • , ei . 

i j j * \ \ . * • , . ^ j A long-term and snort-term statistical profile can be 

load data). These measures can detect a wide variety of data , ~ c . . 

transfer techniques such as a large volume of sdull data 8 enerated f °f ^ent stream. Thus, different event 

transfers via e-mail or downloading targe files en masse. The streams can slice network packet data m different ways, 

monitor may distinguish between network packer based on 10 For « xam P le > » ™>* **** ""X only network 

the time such packets were received by the network entity, P^kets having a source address corresponding to a satellite 

permitting statistical analysis to distinguish between a nor- office - ™f> a u lo0 ^ «nd short-term profile will be 

mal data transfer during a workday and an abnormal data S e ( n ™ ted £ r the P ut ™ lu «tollito office. Thus, althougti , a 

transfer on a weekend evening. c satellite office may have more privileges and should be 

, . . , 15 expected to use more system resources than other external 

Attempted network intrusion may also produce anoma- ,\ fll * . «. . . . „ 

, , \ _ ^ , J . , i . . addresses, a profile of satellite office use can detect address 

lous levels of errors. For example, categorical and intensity spoofing - (i . e modifying packe t information to have a 

measures derived from pnvilegc errors may indicate of ^ Qffice) 

attempts to access protected files, directories, or other net- , , 

work assets. Of course, privilege errors occur during normal 20 The same network packet event may produce records in 

network operation as users mistype commands or attempt to more lhan 0De eve 1 nt strea m J* r sample, one event stream 

perform an operation unknowingly prohibited. By compar- m ^ momtor P ackets for ^ commands while another 

ing the long-term and short-term statistical profiles, a moni- e u vem stream monitors packets from a particular address. In 

tor can distinguish between normal error levels and levels mis case > aa ^command from the address would produce 

indicative of intrusion without burdening a network admin- 25 an event record in each stream - 

istrator with the task of arbitrarily setting an unvarying Referring to FIG. 5, a monitor may also "deinterleave." 

threshold. Other measures based on errors, such as codes me monitor may create and update 74, 76 more than 

describing why a network entity rejected a network packet oae short-term profile for comparison 78 against a single 

enable a momtor to detect attempts to infiltrate a network long-term profile by identifying one of the multiple short- 

with suspicious packets. 30 term P r °nles tnat be updated by an event record in an 

Attempted network intrusion can also be detected by 3 ° event stream. For example, at any one time a network entity 

measures derived from network connection information. For mav handle several ^ "anonymous" sessions. If each 

example, a measure may be formed from the correlation network P acket for aU anonymous sessions were placed in a 

(e.g., a ratio or a difference) of the number of SYN connec- sm &* short-term statistical profile, potentially intrusive 

tion request messages with the number of SYN_ACK 35 activilv of one anonymous session may be statistically 

connection acknowledgment messages and/or the number of ameliorated by non-intrusive sessions. By creating and 

ICMP messages sent. Generally, SYN requests received updating short-term statistical profiles for each anonymous 

should balance with respect to the total of SYN_ACK and session ' each anonymous session can be compared against 

ICMP messages sent. That is, flow into and out-of a network me long-term profile of a normal FTP anonymous session, 

entity should be conserved. An imbalance can indicate 40 Demte^ving can be done for a vmety of sessions includ- 

repeated unsuccessful attempts to connect with a system, m S HTrP sessions (e.g., a short-term profile for each 

perhaps corresponding to a methodical search for an entry browser session). 

point to a system. Alternatively, intensity measures of Referring to FIG. 6, a computer ^ platform 14 suitable for 

transport-layer connection requests, such as a volume analy- executing a network momtor 16 includes a display 50, a 

sis of SYN-RST messages, could indicate the occurrence of 45 keyboard 54, a pointing device 58 such as a mouse, and a 

a SYN-attack against port availability or possibly port- digital computer 56. The digital computer 56 includes 

scanning. Variants of this can include intensity measures of memory 62, a processor 60, a mass storage device 64a, and 

TCP/FIN messages, considered a more stealthy form of port otner customary components such as a memory bus and 

scanning. peripheral bus. The platform 14 may further include a 

Many other measures can detect network intrusion. For 50 networ ^ connection 52. 

example, "doorknob rattling," testing a variety of potentially Mass storage device 64a can store instructions that form 

valid commands to gain access (e.g., trying to access a a monitor 16. The instructions may be transferred to memory 

"system" account with a password of "system"), can be 62 and processor 60 in the course of operation. The instruc- 

detected by a variety of categorical measures. A categorical aons 16 can cause the display 50 to display images via an 

measure of commands included in network packets can 55 interface such as a graphical user interface. Of course, 

identify an unusual short-term set of commands indicative of instructions may be stored on a variety of mass storage 

"doorknob-rattling." Similarly, a categorical measure of devices such as a floppy disk 646, CD-ROM 64c, or PROM 

protocol requests may also detect an unlikely mix of such (not shown). 

requests. Other embodiments are within the scope of the following 

Measures of network packet volume can also help detect 60 claims, 

malicious traffic, such as traffic intended to cause service What is claimed is: 

denials or perform intelligence gathering, where such traffic 1- A method of network surveillance, comprising: 

may not necessarily be violating filtering policies. A mea- receiving network packets handled by a network entity; 

sure reflecting a sharp increase in the overall volume of building at least one long-term and at least one short-term 

discarded packets as well as a measure analyzing the dis- 65 statistical profile from at least one measure of the 

position of the discarded packets can provide insight into network packets, the at least one measure monitoring 

unintentionally malformed packets resulting from poor line data transfers, errors, or network connections; 
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comparing at least one long-term and at least one short- 
term statistical profile; and 

determining whether the difference between the short- 
term statistical profile and the long-term statistical 
profile indicates suspicious network activity. 5 

2. The method of claim 1, wherein the measure monitors 
data transfers by monitoring network packet data transfer 
commands. 

3. The method of claim 1, wherein the measure monitors 
data transfers by monitoring network packet data transfer 10 
errors. 

4. The method of claim 1, wherein the measure monitors 
data transfers by monitoring network packet data transfer 
volume. 

5. The method of claim 1, wherein the measure monitors 15 
network connections by monitoring network connection 
requests. 

6. The method of claim 1, wherein the measure monitors 
network connections by monitoring network connection 
denials. 20 

7. The method of claim 1, wherein the measure monitors 
network connections by monitoring a correlation of network 
connections requests and network connection denials. 

8. The method of claim 1, wherein the measure monitors 
errors by monitoring error codes included in a network 25 
packet. 

9. The method of claim 8, wherein an error code com- 
prises a privilege error code. 

10. The method of claim 8, wherein an error code com- 
prises an error code indicating a reason a packet was 30 
rejected. 

11. The method of claim 1, further comprising responding 
based on the determining whether the difference between the 
short-term statistical profile and the long-term statistical 
profile indicates suspicious network activity. 35 

12. The method of claim 11, wherein responding com- 
prises transmitting an event record to a network monitor. 

13. The method of claim 12, wherein transmitting the 
event record to a network monitor comprises transmitting 
the event record to a hierarchically higher network monitor. 40 

14. The method of claim 13, wherein transmitting the 
event record to a network monitor comprises transmitting 
the event record to a network monitor that receives event 
records from multiple network monitors. 

15. The method of claim 14, wherein the monitor that 45 
receives event records from multiple network monitors 
comprises a network monitor that correlates activity in the 
multiple network monitors based on the received event 
records. 

16. The method of claim 11, wherein responding com- 50 
prises altering analysis of the network packets. 

17. The method of claim 11, wherein responding com- 
prises severing a communication channel. 



18. The method of claim 1, wherein the network packets 
comprise TCP/IP packets. 

19. The method of claim 1, wherein the network entity 
comprises a gateway, a router, or a proxy server. 

20. The method of claim 1, wherein the network entity 
comprises a virtual private network entity. 

21. A method of network surveillance, comprising: 
monitoring network packets handled by a network entity; 
building a long-term and multiple short-term statistical 

profiles of the network packets; 

comparing one of the multiple short-term statistical pro- 
files with the long-term statistical profile; and 

determining whether the difference between the one of the 
multiple short-term statistical profiles and the long- 
term statistical profile indicates suspicious network 
activity. 

22. The method of claim 21, wherein the multiple short- 
term statistical profiles comprise profiles that monitor dif- 
ferent anonymous FTP sessions. 

23. The method of claim 21, wherein building multiple 
short-term statistical profiles comprises deinterleaving pack- 
ets to identify a short-term statistical profile. 

24. A computer program product, disposed on a computer 
readable medium, the product including instructions for 
causing a processor to: 

receive network packets handled by a network entity; 

build at least one long-term and at least one short-term 
statistical profile from at Least one measure of the 
network packets, the measure monitoring data 
transfers, errors, or network connections; 

compare at least one short-term and at least one long-term 
statistical profile; and 

determine whether the difference between the short-term 
statistical profile and the long-term statistical profile 
indicates suspicious network activity. 

25. A method of network surveillance, comprising: 
receiving packets at a virtual private network entity; and 
building at least one long-term and at least one short-term 

statistical profile based on the received packets, and 
comparing at least one long-term statistical profile with at 
least one short-term statistical profile to determine 
whether the packets indicate suspicious network activ- 
ity. 

26. The method of claim 25, further comprising decrypt- 
ing the packets before statistically analyzing the packets. 

27. The method of claim 25, further comprising not 
decrypting the packets before statistically analyzing the 
packets. 
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